10 - 601 B Recitation 2

In this problem we illustrate how gradients can be used to solve the least squares problem. Suppose we have input data matrix X ∈ Rn×p, output data y ∈ R and weight vector w ∈ R, where p is the number of features per observation. The linear system Xw = y corresponds to choosing a weight vector w that perfectly predicts yi given X{i, :} for all observations i = 1, . . . , n. The least squares problem arises out of the setting where the linear system Xw = y is overdetermined, and therefore has no solution. This frequently occurs when the number of observations is greater than the number of features. This means that the outputs in y cannot be written exactly in terms of the inputs X. So instead we do the best we can by solving the least squares problem: